Skip to content

OPENJPA-2940 - Jakarta Persistence 3.2 #142

Open
rzo1 wants to merge 195 commits intoapache:masterfrom
rzo1:OPENJPA-2940
Open

OPENJPA-2940 - Jakarta Persistence 3.2 #142
rzo1 wants to merge 195 commits intoapache:masterfrom
rzo1:OPENJPA-2940

Conversation

@rzo1
Copy link
Copy Markdown

@rzo1 rzo1 commented Mar 30, 2026

Hi all, @solomax, @cristof, all

This PR builds on @cristof's JPA 3.2 work, completed with the assistance of generative AI (Claude).

It fully passes the JPA 3.2 TCK (at least on PostgreSQL).

<?xml version="1.0" encoding="UTF-8"?>
<failsafe-summary xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/failsafe-summary.xsd" result="null" timeout="false">
    <completed>2135</completed>
    <errors>0</errors>
    <failures>0</failures>
    <skipped>4</skipped>
    <failureMessage xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"/>
</failsafe-summary>

Tests run on Java 21, though there are some local failures - similar to what I see on master, likely an OSX or Java 21 issue rather than something introduced here:

  • org.apache.openjpa.persistence.detach.TestDetachNoProxy: 2 failures (testClear10Compat, testClear20Compat) with a proxy subclass issue
  • org.apache.openjpa.event.kubernetes.KubernetesTCPRemoteCommitProviderTest: 1 failure (addresses)

The PR is fairly large, so feel free to cherry-pick whatever is useful. I expect there are concerns and things to address before this is merge-ready.

A number of new tests have been added that mirror TCK behavior. Happy to drop those if they're unwanted.

That said, it was a fun three-week experiment :)

solomax and others added 30 commits July 8, 2025 09:13
* Updated tentative version to 4.2.0-SNAPSHOT
* Updated java version to 17
* Updating project to exclude almost all java.security deprecated calls
* Passes default profile tests, but fails with postgres
* Fixed some non-deterministic tests that fail with postgresql
* Tested and passed XML support using postgresql-17 as target db
* Replacing string number constructors
* Removing dangling SecurityContext references
* removed TestSecurityContext because it is terminally deprecated since 17 and already removed in current JDK versions
* updated h2-2 test profile jdbc url to remove strict definition
* updated openjpa-slice and openjpa-xmlstore pom system variables definitions
* updated GH actions workflows to use test-h2-2 profiles
* Project passes tests on derby, h2-2, postgres:latest, mysql:lts,
mariadb:lts
* Updated dependency version
* Added API new methods to API implementation classes with methods that
throw UnsupportedOperationException, except for four methods in
EntityManagerImpl that required proper implementations to pass tests
* Project is still passing tests on derby and postgresql, at least
* Added XML JPA 3.2 schema and definitions
* Added configuration support by 3.2 version
* Added SchemaManager impl and corresponding methods in BrokerFactory
interface
* Added concrete working (not for h2-2) implementation of SchemaManager
methods for JDBCBrokerFactory
* Added concrete working impl for EMF#getName()
* Reverting unnecessary changes
* Fixing broken map synchronization
* Changing signature of BrokerFactory API on schema dealing validate method
* Adding test to check if validate operation throws exception when it fails
* Changing GH CI workflow to allow usage of both self-hosted and GH hosted runners
* Tested on derby, h2-2, postgresql:latest, mysql:lts, mariadb:lts
* Implementing emf creation passing PersistenceConfiguration
* Removing unused import in BrokerImpl
* Implemented new PersistenceUnitUtil load methods
* Moved PUU loading tests to test unit already present
* Updated test unit to junit 4.x format
rzo1 added 15 commits April 21, 2026 09:09
TestSchemaGenAnnotations and TestSchemaGenerationScripts hardcoded ANSI
ADD CONSTRAINT ... FOREIGN KEY / DROP CONSTRAINT syntax, but MariaDB and
MySQL legitimately emit the equivalent ADD FOREIGN KEY <name> / DROP FOREIGN
KEY <name> form (constraintNameMode = CONS_NAME_MID, intentional in the
MySQL/MariaDB dictionaries). Accept either form in the assertions.
The jpql.treatjoinon.TCustomer entity used @table(name="TCUSTOMER") which
collides with xmlmapping.entities.Customer (composite EmbeddedId with NOT NULL
countryCode). On shared-DB test infrastructures (MariaDB, PostgreSQL), this
caused schema drop/recreate cycles to fail with 'ALTER TABLE TCUSTOMER DROP
COLUMN id' since id is a PK in the xmlmapping schema.

Rename to TTREAT_CUSTOMER to isolate the tables.
MariaDB Connector/J 3.5.4 reports per-row failure indexes correctly (unlike
older Oracle/older drivers that flag the first row of the batch). A prior
commit mistakenly grouped MariaDB with Oracle/Postgres in the 'first-row
reported' assertion branch. Use the generic per-row assertion for MariaDB.
Instant, LocalDateTime, Date, and Calendar version columns left column
precision unset, producing DATETIME(0) on MariaDB/MySQL by default. Two
updates within the same whole second stored identical versions, defeating
optimistic locking. Request 6 fractional digits (microseconds) in
ColumnVersionStrategy.map() for temporal javaTypes when the user has not
already specified precision. DATETIME(6) / TIMESTAMP(6) matches what
Derby/PostgreSQL already use.
…B/MySQL

MariaDB >= 10.2 and MySQL >= 5.7 support fractional-second precision on
DATETIME/TIME columns. Bump dateFractionDigits to 6 inside the existing
version-gated block so temporal columns use DATETIME(6)/TIME(6) by default.

This fixes optimistic-locking on @Version Instant/LocalDateTime columns
where two updates within the same whole second otherwise stored identical
versions. Matches PostgreSQL and Derby defaults.
…r.proxy

SingleFieldManager.proxy assumed the datastore value for a @TeMPOraL field
always matches the field's declared type. Handler strategies (and some
JDBC driver paths) can return a Calendar for a java.util.Date field or a
Date for a Calendar field, which caused a ClassCastException at runtime
(e.g. TCK StoredProcedureQuery Client1 where a GregorianCalendar proxy
was assigned to an Employee.hireDate Date field).

Coerce between Calendar and Date on store so the proxy receives a value
of the correct type regardless of which the handler produced.

Also fix a copy/paste bug in ProxyManagerImpl.generateAndLoadProxyCalendar
which passed ProxyDate.class as the ancestor class used to resolve the
proxy class loader's package/protection domain.
…Connection when tx is active

Per JPA 3.2, the Connection handed to ConnectionConsumer/ConnectionFunction is
on loan to the user code; its lifecycle belongs to the EntityManager/Broker
when a transaction is active. Previously, the finally blocks in both
EntityManagerImpl.runWithConnection and callWithConnection unconditionally
closed the returned wrapper (a ClientConnection around the broker's
RefCountConnection).

On most drivers (Derby, H2, PostgreSQL) the close path is benign because
RefCountConnection._retain is true during an active tx so free() is not
invoked. MariaDB Connector/J 3.5.4, however, reacts more aggressively when
its connection wrapper is closed inside an active transaction and discards
pending work on the underlying connection, causing manual INSERTs issued
from within runWithConnection to be lost. This manifests as
TestCascadeManyToOneAndEM2.testRunWithConnection / testCallWithConnection
failing on MariaDB with "Order should be found ...".

Fix: only close the user-facing connection when no transaction is active,
i.e. when the user borrowed a free-standing connection. During an active
transaction the EM/broker retains ownership of the connection.
MySQL and MariaDB don't support ANSI 'NULLS FIRST' / 'NULLS LAST' on
ORDER BY. Route the ordering-suffix emission through
DBDictionary.appendNullsPrecedence so dialects can customize.

Default implementation emits 'NULLS FIRST' / 'NULLS LAST' unchanged.
MySQLDictionary and MariaDBDictionary override to emulate via the
'<expr> IS NULL <sort>' auxiliary sort key. The emulation is skipped
when MySQL's default NULL ordering (NULLs first for ASC, NULLs last for
DESC) already matches the requested precedence.
…DK 24+

Ant's non-forked <java> task installs a SecurityManager to catch
System.exit(), but System.setSecurityManager() throws
UnsupportedOperationException on JDK 24+. The generator would fail
silently (no failonerror), producing no build-time proxy classes and
causing TestDetachNoProxy legacy-compat assertions to fail because
runtime proxies are never tagged detachable=true.

Add fork=true and failonerror=true so the generator runs in a
separate JVM and surfaces any errors.
…r config

- docker-compose-test-pg17.yml (PostgreSQL 17)
- docker-compose-test-mariadb.yml (MariaDB 11.4)
- docker-compose-test-mysql.yml (MySQL 8.4)

The test-mysql-docker profile still used the obsolete mysql:mysql-connector-java
GAV and com.mysql.jdbc.Driver class name. Update to com.mysql:mysql-connector-j
and com.mysql.cj.jdbc.Driver so modern MySQL Connector/J (9.x) resolves and
connects.
MySQL stores BOOLEAN columns as TINYINT(1). By default, MySQL Connector/J
reports them as Types.BIT, which OpenJPA's MappingInfo.mergeColumn
coerces as "numeric" and silently upgrades to VARCHAR — defeating
SchemaManager.validate() detection of a genuine VARCHAR->BOOLEAN schema
drift (TestJDBCSchemaManager.testValidate).

Setting transformedBitIsBoolean=true makes Connector/J report TINYINT(1)
as Types.BOOLEAN, which isNumericType() excludes. This matches MariaDB
Connector/J's default behavior where the same test already passes.
Two PostgreSQL-specific fixes needed to make TestJDBCSchemaManager pass:

1. PostgreSQL's JDBC driver reports native boolean columns as Types.BIT.
   OpenJPA's generic column-type comparison treats BIT as numeric and
   silently coerces any VARCHAR mapping to 'upgrade' such columns to
   VARCHAR, which defeats SchemaManager.validate() detection of a
   genuine VARCHAR -> BOOLEAN schema drift. Override newColumn() to
   translate reflected columns whose native type name is 'bool' or
   'boolean' back to Types.BOOLEAN so the type comparison treats them
   as their true logical type.

2. IDENTITY-column sequences are dropped implicitly with their owning
   table, but OpenJPA's schema reflection may still list the sequence
   as extant. Override getDropSequenceSQL to emit
   'DROP SEQUENCE IF EXISTS' so refresh / truncate actions are
   idempotent and don't fail with 'sequence "..." does not exist'
   on PostgreSQL 17.
Truncate is a best-effort operation - the repo SchemaGroup may declare
tables the dialect never physically created (e.g. OPENJPA_SEQUENCE_TABLE
on dialects that prefer native sequences, or tables owned by generators
that were never exercised, or tables qualified by a schema that's not on
the connection's current search_path).

Set _ignoreErrs around the DELETE FROM statements so individual missing
tables are logged as warnings instead of aborting the whole truncate.
Fixes TestJDBCSchemaManager.testTruncate on PostgreSQL 17.
…testing

MariaDB uses 3306. Putting MySQL on 3307 externally (still 3306 inside the
container) lets both run simultaneously on the same host, enabling parallel
cross-DB test runs.
@rzo1
Copy link
Copy Markdown
Author

rzo1 commented Apr 21, 2026

Did look into MariaDB / MySQL and the local build on my OSX looks good. Maybe some one can some additional runs for the tests?

61e93dd might be worth a direct port to main since it fixes a regression on Java 25+ ;-)

@cristof
Copy link
Copy Markdown
Contributor

cristof commented Apr 21, 2026

@rzo1, thanks for the effort. I'll run on mariadb, mysql and pg-18 tomorrow. As soon I have some results, I'll post here.

@cristof
Copy link
Copy Markdown
Contributor

cristof commented Apr 23, 2026

@rzo1, in a clean build, using jdk17 (minimum java version per spec), pointed to PG-17 and MariaDB. PG tests passed persistence jdbc, but failed at locking tests (TestPessimisticLocks::testQueryAfterFindWithPessimisticLocks).
MariaDB LTS (11.8) had 4 errors on persistence JDBC - all of them into TestBatchLimitException.
How do you propose we can finish the work?
Do you have a docker available to reproduce the tests locally?

I startup the pg docker using

docker run --rm --name openjpa-pg -e POSTGRES_PASSWORD=SOME_PWD postgres:17

And create the test database afterwards (docker exec -it openjpa-pg createdb openjpa).
MariaDB is started using:

docker run --rm --name openjpa-mariadb -e MARIADB_USER=openjpa -e MARIADB_PASSWORD=SOME_PWD -e MARIADB_DATABASE=openjpa -e MARIADB_ROOT_PASSWORD=passwd mariadb:lts

The tests are run using, from the directory where I've pulled the branch and provided a uid/gid equals to 1000:1000:

docker run --rm --name openjpa -u 1000:1000 -v ~/.m2:/opt/openjpa/.m2 -v $(pwd):/opt/openjpa -e MAVEN_CONFIG=/opt/openjpa/.m2 -w /opt/openjpa maven:3-eclipse-temurin-17 mvn -Duser.home=/opt/openjpa -Ptest-mariadb -Dopenjpa.mariadb.url=jdbc:mariadb://<DOCKER_DB_ADDRESS>/openjpa -Dopenjpa.mariadb.username=openjpa -Dopenjpa.mariadb.password=SOME_PWD clean package

docker run --rm --name openjpa -u 1000:1000 -v ~/.m2:/opt/openjpa/.m2 -v $(pwd):/opt/openjpa -e MAVEN_CONFIG=/opt/openjpa/.m2 -w /opt/openjpa maven:3-eclipse-temurin-17 mvn -Duser.home=/opt/openjpa -Ptest-postgresql -Dopenjpa.postgresql.url=jdbc:postgresql://<DOCKER_DB_ADDRESS>/openjpa -Dopenjpa.postgresql.username=postgres -Dopenjpa.postgresql.password=SOME_PWD clean package

@rzo1
Copy link
Copy Markdown
Author

rzo1 commented Apr 23, 2026

I am on OSX and using the committed docker-compose.yaml files to run the databases from my dev setup (so I actually have an IDE to debug :D ) - can also run in docker, but will take a few days until I can come back to it :)

Although it shouldn't matter if the docker containers are managed via docker-compose or native. Most likely it is related to the Java version, so I will go down to testing with Java 17.

rzo1 added 3 commits April 23, 2026 14:48
…orts on MariaDB

MariaDB changed batch-failure reporting semantics between server 11.4
and 11.8 (with Connector/J 3.5.4 unchanged): 11.4 reports the actual
duplicate row, 11.8 reports the first row of the failing batch, same
as Oracle/Postgres. Commit 83e933a assumed per-row was the future
behaviour and removed the MariaDB branch; that assumption does not
hold on 11.8.

Restore the isMariaDB flag and broaden the MariaDB assertion to accept
either outcome, using the same pattern already in place for Oracle 18.
Verified green against MariaDB 11.4 and 11.8 under Connector/J 3.5.4.
mariadb:lts now resolves to 11.8 while our compose was pinned to 11.4,
causing drift between local runs and what reviewers / CI pull with the
lts tag. Pin to 11.8 explicitly so TestBatchLimitException and
friends exercise the current LTS behaviour by default.
MySQL 8.4 defaults users to caching_sha2_password. Connector/J refuses
to fetch the server's RSA public key over a non-TLS (useSSL=false)
connection unless allowPublicKeyRetrieval=true is set, so every test
currently fails with "Public Key Retrieval is not allowed". Add the
flag to the test URL so test-mysql-docker works out of the box against
the mysql:8.4 image we ship in docker-compose-test-mysql.yml.
@rzo1
Copy link
Copy Markdown
Author

rzo1 commented Apr 23, 2026

@cristof I am unable to reproduce TestPessimisticLocks::testQueryAfterFindWithPessimisticLocks locally. Pushed a fix for the BatchException thing, which is a handling thing of the jdbc driver used:

[INFO] OpenJPA Parent POM ................................. SUCCESS [  0.638 s]
[INFO] OpenJPA Utilities Library .......................... SUCCESS [  7.408 s]
[INFO] OpenJPA Kernel ..................................... SUCCESS [  5.252 s]
[INFO] OpenJPA JDBC ....................................... SUCCESS [  4.151 s]
[INFO] OpenJPA Persistence ................................ SUCCESS [  2.457 s]
[INFO] OpenJPA Persistence JDBC ........................... SUCCESS [07:21 min]
[INFO] OpenJPA Persistence Locking Tests .................. SUCCESS [18:10 min]
[INFO] OpenJPA tools ...................................... SUCCESS [  0.031 s]
[INFO] OpenJPA Maven Plugin ............................... SUCCESS [  2.175 s]
[INFO] OpenJPA XML Store .................................. SUCCESS [  1.057 s]
[INFO] OpenJPA Slice ...................................... SUCCESS [  0.575 s]
[INFO] OpenJPA JEST ....................................... SUCCESS [  1.202 s]
[INFO] OpenJPA Kubernetes ................................. SUCCESS [  1.302 s]
[INFO] OpenJPA Aggregate Jar .............................. SUCCESS [  1.953 s]
[INFO] OpenJPA Aggregate Jar with Dependencies ............ SUCCESS [  0.449 s]
[INFO] OpenJPA Project Docs and Assemblies ................ SUCCESS [  5.994 s]
[INFO] OpenJPA Examples ................................... SUCCESS [  0.036 s]
[INFO] OpenJPA Examples - Simple .......................... SUCCESS [  0.133 s]
[INFO] OpenJPA Examples - image-gallery ................... SUCCESS [  0.914 s]

@rzo1
Copy link
Copy Markdown
Author

rzo1 commented Apr 23, 2026

Do we have ASF CI set-up for OpenJPA, so we can run on ASF Infra to see results against current main?

@cristof
Copy link
Copy Markdown
Contributor

cristof commented Apr 23, 2026

Yes, we have. I've altered former openjpa-pr-17 build config to openjpa-JPA-3.2 and changed default build branch to OPENJPA-2940 (https://ci-builds.apache.org/job/OpenJPA/job/openjpa-JPA-3.2/).
The build is being made now using profile test-h2-2. Do you want to use another profile for that?

@rzo1
Copy link
Copy Markdown
Author

rzo1 commented Apr 23, 2026

https://ci-builds.apache.org/job/OpenJPA/job/openjpa-JPA-3.2/

Alright thx - yeah I thought we might have the other DBMS via ASF CI as well.

@rzo1
Copy link
Copy Markdown
Author

rzo1 commented Apr 23, 2026

https://ci-builds.apache.org/job/OpenJPA/job/openjpa-JPA-3.2/

Alright thx - yeah I thought we might have the other DBMS via ASF CI as well.

Think we won't see anything meaningful since my changes are not on the ASF Branch (but on my fork); perhaps you need to resync the current ASF OPENNLP-2940 branch with the changes from my fork.

@cristof
Copy link
Copy Markdown
Contributor

cristof commented Apr 23, 2026

I've pushed a new branch so CI can see your PR (OPENJPA-2940-rzo1). Started a build in ci-builds (still -Ptest-h2-2).

@rzo1
Copy link
Copy Markdown
Author

rzo1 commented Apr 24, 2026

I now have a university vm running Ubuntu 24.04 available and can confirm that PG17/PG18 unit tests are broken running under Linux (works on OSX though). Goging to remote-debug via that VM now because seems a test order thingy ;-) - stay tuned.

…build matrix

On PostgreSQL 17+, binding a Java boolean into a SMALLINT column is
rejected without an explicit cast. The create-postgresql.sql script
declared CUSTMAPPC.FEMALE as SMALLINT, which leaked into the DB when
TestSchemaGenerationProperties.testSchemaGenScriptCreate ran before
TestEJBCustomMapping (filesystem-order dependent). Change FEMALE to
BOOLEAN so the script matches what PostgresDictionary would produce at
runtime (bitTypeName="BOOL").

Also add scripts/run-build-matrix.sh, which drives the dockerised
mariadb/mysql/pg17/pg18 matrix. It exports TZ=UTC before invoking
Maven so the JVM and the docker DB sessions agree on wall clock.
This prevents TestEJBQLFunction.testExtractHourFromLocalTime from
flaking when the host JVM default TZ differs from the PG session TZ
(e.g. Europe/Berlin host vs UTC container).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants